@dang I feel like this is getting close to the lowest level of discourse and occurs somewhat often these days where people overly reference the output of some AI and then are challenging the results without googling. I feel a proper hacker ethos would spur someone to find a real answer. So in honor of the name of the site maybe disallow refutations or other dismissals of a submission based on the output of an AI or overly long discussion that is offtopic about an AI?
Tom and I of course agree! But it's not so easy to disallow things, since humans do $thing anyway.
Most probably the long-term solution is for the community to express this preference by flagging comments that do this. I think we see signs of that happening, which is good, because a community solution is better than a technical solution (and way better than an authority solution) any day.
Btw, tapirl is a good contributor and I don't think they did anything particularly wrong. It just takes time for us all to learn the optimal patterns together.
Sorry I read about this way long ago, what is meant by "worked for me" "project", "behind my back" and such. I didn't get the connotations for most of that from reading articles about this. What is the real story?
I have been thinking a bit lately about how much sense that makes compared to just using word vectors, since traditional queries are super short and often keyword based(like searching for "ground beef" when wanting "ground beef recipes I can cook easily tonight") and so lack most of the context that BERT or similar gives you. I know there are methods like using seperate embeddings for queries and such, but maybe a basic word based search could be more useful, especially with something like fastText for out of vocabulary terms.
There is a bunch of new phrases people have been using around AI topics, so it can be hard to tell what exactly is being talked about, like "Roko's Basilisk", the "lottery card hypothesis?"(not quite remembering the phrasing on this one), "the bitter lesson", "paperclip maximizing", "stochastic parrot", etc.. Thank you for clarifying. I was kind of hoping there was a fun blog or story with a "banana zone".
I usually exercise ~5 days a week and on those days will shower or bathe twice. On days when I am doing yard work or grilling it might bathe 3 times depending on how poorly I planned out my day.
Sorry if I missed it, but what about keeping the suffixes and trying to do some finetuning on the source then clustering sentences or at least pages which given the media should be consistent-ish
Great question — and something I've been thinking about. I stripped suffixes mostly to normalize some of the repeated endings (aiin, dy, etc.) that felt like filler, but you’re totally right that preserving them might preserve structure I lost.
Clustering by sentence or page would be interesting too — I haven't gone that far yet, but it’d be fascinating to see if there’s consistency across visual/media sections. Appreciate the insight!
How does it handle documents longer than the context length of the model? Sorry there are a ton of these regularly and they don't usually think about this.
Edit: it seems like it just splits in to sentences which is a weird thing to do given in English only 95%ish percent agreement is even possible on what a sentence is.
```
// Process in batches
for batch in sentences.chunks(batch_size) {
// Truncate each sentence to max_length * median_token_length chars
let truncated: Vec<&str> = batch
.iter()
.map(|text| {
if let Some(max_tok) = max_length {
Self::truncate_str(text, max_tok, self.median_token_length)
} else {
text.as_str()
}
})
.collect();
```
Sorry, looking more, it doesn't seem like you are doing what you are saying. This is just poorly breaking text into bad chunks with no regard for semantics and is like ~200 lines of actual code. What is this for? Most models can handle fairly large contexts.
Edit: That wasn't intended to be mean, although it may come off that way, but what is this supposed to be for? Myself I have text >8k tokens that need to be embedded and test things regularly.
I think you are referring to for "batch in sentences.chunks(batch_size)"? This is not actually chunking sentences, chunks() is simply an iterator over a slice (in this case, a slice of all our input sentences of length batch_size). We don't have an actual constraint on input length. We truncate to 512 tokens by default, but you can easily set that to any amount by directly calling encode_with_args. There's an example in our quickstart: https://github.com/MinishLab/model2vec-rs/tree/main?tab=read....
Sorry if this comment is outdated or ill-informed, but it is hard to follow the current news. Do the Phi models still have issues with training on the test set, or have they fixed that?
So natural is a stretch at the moment, but you can use all kinds of different techniques, what is needed is more community and library standardization around some solutions.
