Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Shower thought: But how often per week do you run the pelican these days? And do you have it automated at this point or would the automation take out the meaning of the benchmark?


My automation is pretty simple. I use my https://llm.datasette.io tool where I have a template saved:

  llm "Generate an SVG of a pelican riding a bicycle" --save pelican
When a model comes out I first make sure LLM can talk to it - usually by updating the relevant plugin, but if it's on OpenRouter I can use it directly with https://github.com/simonw/llm-openrouter - sometimes I use this mechanism instead, for OpenAI-compliant API models: https://llm.datasette.io/en/stable/other-models.html#configu...

Then I run something like this:

  llm -m gpt-6-astra -m pelican
Then I grab the most recent log export as markdown:

  llm logs -cu | pbcopy
-c means most recent conversation, -u includes token usage

I paste that into https://gist.github.com and then paste the resulting Gist URL into the URL tab on https://tools.simonwillison.net/markdown-svg-renderer

If the model supports multiple reasoning levels I run it once per level and put those in the same file.

I really should automate this a bit more.




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: