Zan wrote:What are the necessary and sufficient conditions for the two numbers $a$ and $b$ so that it is possible to form a rectangle in a Cartesian coordinate system which does not contain points with integer coordinates?
I am modifying your problem so that we can find an answer. the way you are looking an answer is not proper. There is no legitimate definition of $a,b$ and so more ambiguities. Let's try to find the vertices of a rectangle which have no point inside it. A point with integer coordinate is called
lattice point. Let's say the rectangle has vertices on integer points but no inner points is lattice.
We prove the following lemma:
Lemma:
The segment $L(A, B)$ with $A(x_1, y_1), B(x_2, y_2)$ contains $\gcd(|x_1-x_2|, |y_1-y_2|)$ lattice points.
Proof:
Let's count the number of points on the line which cuts the $X$-axis in $(a,0)$ and the $Y$-axis in $(0,b)$. Then it has the equation, \[\frac xa+\frac yb=1\]
We need to count the number of integer solutions to this equation. Say, $\gcd(a,b)=g,a=gm,b=gn$ with $\gcd(m,n)=1$. Then the equation reduces to, \[mx+ny=gmn\]
$m$ divides $gmn$ and $mx$, so $m$ divides $ny$ but $n$ and $m$ share no common factor. Therefore, $m|y$ and similarly $n|x$. Say, $x=mt, y=nu$. We have $t+u=g$. This equation has $g+1$ solutions. So, there will be total $g+1$ points including the origin. To count the number of lattice points on $L$, shift the origin to $(x_1,y_1)$. At this, the number of lattice points will be the same, even after rotation, it will be same.
Now, say the rectangle has vertices $(x_i,y_i)_{i=1}^4$. So, now the number of lattice point is equal to $\gcd(x_2-x_1,y_2-y_1)$. We define $G_i=\gcd(|x_i-x_{i+1}|, |y_i-y_{i+1}|)$ with $(x_{n+1}, y_{n+1})=(x_1,y_1)$. The number of lattice points on the rectangle will be(of-course you can extend it to any polynomial also) \[G=\sum_{i=1}^4G_i\]
This primitive rectangle must have an area of $1$. From Pick's theorem, \[A=I+\frac B2-1\]
\[1=0+\frac G2-1\Rightarrow G=4\Rightarrow\sum_{i=1}^4G_i=4\]
Since $G_i>0$, we can assume the solution, $G_1=G_2=G_3=G_4=1$. Again, the Farey sequence comes. Take four of them so that they make a rectangle. I mean you should consider a point from the sequence as $(y,x)$ if the fraction in the sequence is $\frac xy$.