If you are comming by bus, you should take bus 65, direction to "Zvezdara 2" and get off on last bus stop.
Closest bus stop to the Serbian Academy of Science and Arts is "Zeleni venac". You can buy bus ticket at the driver, but be aware that you should prepare exact amount of money for the ticket, since there can be a problem if they don't have the change. Bus ticket (in one direction) is 150 dinars.
From the last bus stop you have to walk to Astronomical Observatory, as shown on next picture.