Checked the wiring for a hundreds of times, tried to modify the code to confirm that the “math” behind does not impact the angle.
Spent several days to troubleshoot, reading documents and even forums to find the reason. Thanks to the fellow hams listed below the suspicion was aimed to the power supply itself. Turned out that in case 5V external power is used (planned to operate the module USB independently) then the internal sampling voltage is only 4.04V. This voltage has direct co-relation with the “distance” of the UDC generation. This is actually because the internal regulator understandably cannot produce stable 5V sampling reference if external 5V is used only for supply.
The solution was to use external reference voltage to the AREF pin which made the sampling complete and the UDC reached 360 degrees even at 1016, so there was even some buffer remaining as well. Interestingly enough, enabling external reference is possible with “analogReference (EXTERNAL)” switch. Even tho it was declared into the code the reading still did not work properly. After couple of iterations it turned out that the switch is not needed, and if the Vin was simply connected to the AREF pin the UDC calculation started to work flawlessly. However this is something against the “science” as without the switch the reference voltage is taken from internal source but I let myself to leave this question unanswered as the azimuth display works as it should.
The next step will be to create the enclosure and finalize the solution.
Thanks for the inspirations, hints and support:
AB7E
DF7TV
HA3FHH
YT2FSG