Unless, of course, the secondary index has all of the columns you need to execute your query. (Sometimes it can help to add an extra column or two to your secondary index if you want to avoid the extra lookup.)
sqlite3 stores rows in an integer primary key index. If you do not specify an "integer primary key" column, it synthesizes one behind the scenes (essentially a rowid) and your primary key lookups end up going through two indexes.
As a side note, because of this, secondary indices just have a pointer to a PK. So when looking up by a secondary index, it has to traverse two trees.