Possibly, yes - Flash and Java are both commonly-exploited attack vectors for this. Not having Java is a good start, but Chrome ships with Flash embedded in it (and to Google's credit, they keep it aggressively patched by pushing updates), but if someone were running an exploit against an unpatched zero-day, then you could go to a harmless site which is running federated ads, which the attacker has purchased ad views for, and which they use to serve their Flash-based payload which runs the exploit and provides some measure of your access to your computer (often to add an additional payload that can be used to back-door you). The damage is done.

To protect yourself against this, you should go into Chrome's about:settings/content and set Plugins to "click-to-play", so that you have to manually allow a plugin to execute, preventing this kind of drive-by attack.