The problem in your solution is a confusion about angle $\theta$ from the cosine rule you are using. The triangle of forces will look like the following image:
where r is the angle in the cosine rule:
$$R^2 = F_1^2 +F_2^2 - 2F_1 F_2 \cos(r)$$
$$R^2 = F_1^2 +F_2^2 - 2F_1 F_2 \cos(180-\theta)$$
If this is superimposed to your initial image it will look like the following:
So what you calculate as 97.9 is actually equal to $180-\theta$. Therefore $\theta$ in your original drawing is $82.096[deg]$.
When you substitute, you can obtain the components of $R_x, R_y$ (notice the use of $\phi$). Therefore:
- $R_x = F_1 + F_2 \cos(\phi) = -868.75[N]$
- $R_y = F_2 \sin(\phi) = 495.25[N]$ (Negative because its pointing to the left)
If you take the magnitude of $R$ you get $\sqrt{(-868.75)^2 + 495.25^2}=1000$ as specified.
Regarding the angle between R (lets denote it $\phi_R$) and the positive X axis:
$$\tan(\phi_R)=\frac{R_y}{R_x} = \frac{495}{-868.75} $$ $$\phi_R=150[deg]$$
Another way
I used the following method to find the angle $\theta$. It leads to the same values.
Lets write $F_1 = -800[N]$ and $F_2=500N$. Also assume the $\phi$ is angle with the positive x, therefore $\phi = 180-\theta$.
Then the resultant R will have:
$$R_x = F_1 + F_2 \cos(\phi) \qquad R_y=F_2 \sin(\phi)$$
Then $$|R|= \sqrt{(F_1 + F_2 \cos(\phi))^2 + (F_2 \sin(\phi))^2 }$$
Solving for this you get
$$\phi= 97.9[deg]\rightarrow \theta = 82.1$$