MoA SynthCode Mix
A mixture-of-agents that synthesizes coding answers from the best LLMs for state-of-the-art performance.
API
Example
import OpenAI from 'openai'const openai = new OpenAI({baseURL: "https://api.crosshatch.app/v1",apiKey: "your_api_key_here"})const completion = await openai.chat.completions({model: "moa-coding",messages: [{role: "user",content: "Hello there"}]})
Models
This mix uses the models below:
claude-3-5-sonnet-20240620
Provided by anthropic
gpt-4-turbo-2024-04-09
Provided by openai
gpt-4o-2024-08-06
Provided by openai
Readme
SynthCode Mix
This is a synthesis mix that uses a mixture-of-agents architecture to give you the highest quality answers. Your request is sent to two "proposer" models (Claude 3.5 Sonnet and GPT-4 Turbo). The responses from these models are passed to an "aggregation" model (GPT-4o) which synthesizes the answers, corrects issues, and returns code.
To learn more about mixture of agents, check out our Github repo here.
Quality
According to our evaluations, this mix performs 18% better than the current leader in Bigcodebench Instruct Hard, an evaluation aimed at measuring the performance of LLMs for difficult coding tasks.
We ran the evaluation on 148 problems in the Bigcodebench Instruct Hard dataset using the provided docker container. Our SynthCode Mix score 31.1% (Pass@1) on this dataset compared to the next best model, GPT-4o, which scored 26.4% (Pass@1).
Performance
This model may take longer to produce a response due to the multiple sub-requests. We are working hard to reduce the latency while retaining the same high quality.
Composition
This mix produces responses from the following models:
| Model Name | Type |
|---|---|
| Claude 3.5 Sonnet | Proposer |
| GPT-4 Turbo | Proposer |
| GPT-4o | Aggregator |