With numerous AI-related open-source projects emerging, the open-source ecosystem is experiencing a true AI open-source renaissance. Starting with the success of LangChain, many open-source projects are quickly filling the gaps in AI industry systems.
Open LLM
LLM (Large Language Model), the core of generative AI, is divided into two axes: Closed LLM led by GPT and Open LLM of the Llama camp. Mistral's team has released their model under an open-source license, and it's garnering a lot of attention due to its exceptional performance. Open LLM is primarily managed and provided through Hugging Face.
**Mixtral-8x7B-Instruct-v0.1** (Apache-2.0)
https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0\.1
- It employs a Mixture-of-Experts (MoE) model.
- It surpasses the Llama 2 70B model and even exhibits performance exceeding GPT-3.5 175B.
- It ranks third in the Chatbot Arena, a blind chatbot test, following GPT-4 and Claude-2.
**Llama-2-7b-chat** (Llama 2 Community)
https://huggingface.co/meta-llama/Llama-2-7b-chat
- It has a license that allows commercial use for services with fewer than 700 million monthly active users.
- Numerous derivative models derived from Llama-2 are available.
**phi-2** (MIT)
https://huggingface.co/microsoft/phi-2
- It's a lightweight model with 2.7B parameters released by Microsoft.
- Test results reportedly indicate that its performance in common sense, language comprehension, and logical reasoning surpasses that of 13B models.
LLM Inference and Serving
To effectively utilize well-trained LLMs, tools are needed that are fast and can efficiently manage computing resources.
**Ollama** (MIT)
https://github.com/jmorganca/ollama
- It allows you to run LLMs of around 7B parameters directly in your local environment (Mac, Linux, Windows).
- You can download and run models with simple commands.
- It enables model management through the CLI and simple chat interactions.
- It offers various functionalities through its provided API.
**vLLM** (Apache-2.0)
https://github.com/vllm-project/vllm
- It's a fast and easy-to-use library for LLM inference and serving.
- It supports models provided by Hugging Face.
- It offers distributed processing, parallel processing, streaming output, and an OpenAI-compatible API.
- It supports Nvidia and AMD GPUs.
**KServe** (Apache-2.0)
https://github.com/kserve/kserve- A platform for ML model inference that can be built in a Kubernetes environment. - It provides an abstraction interface for scaling, networking, and monitoring.
LLM Proxying
**LiteLLM** (MIT)
https://github.com/BerriAI/litellm
- It integrates various LLM APIs and provides a proxy.
- It follows the API format of OpenAI.
- It offers per-user API authentication management.
**One API** (MIT)
https://github.com/songquanpeng/one-api
- It enables immediate access to all large models through a standard OpenAI API format.
- It supports a variety of LLMs and also provides a proxy service.
- It allows for load balancing and multi-deployment, and offers user management and group functionalities.
**AI Gateway** (MIT)
https://github.com/Portkey-AI/gateway
- It provides connections to over 100 LLMs through a single, fast, and familiar API.
- It guarantees fast access with a small installation size.
**LLM Monitoring Great Expectations** (Apache-2.0)
https://github.com/great-expectations/great_expectations
- It helps data teams build a shared understanding of their data through quality testing, documentation, and profiling.
- It integrates with CI/CD pipelines, allowing you to add data quality exactly where it's needed.
**LangFuse** (MIT)
https://github.com/langfuse/langfuse
- It provides open-source LLM visibility, analysis, rapid management, evaluation, testing, monitoring, logging, and tracing.
- You can browse and debug complex logs and traces from a visual UI.
- Enterprise features are planned for the future.
**Giskard** (Apache-2.0, Dual License)
https://github.com/Giskard-AI/giskard
- It can automatically detect vulnerabilities in AI models, from tabular format models to LLMs, such as bias, data leakage, spurious correlations, hallucinations, toxicity, and security issues.
- It supports the quality assurance process for ML models and LLMs by scanning and automatically generating test suites for AI model vulnerabilities.
- It offers a SaaS platform (premium) for detecting AI safety risks in deployed LLM applications.
LLM Framework
LangChain (MIT)
https://github.com/langchain-ai/langchain
- It's a framework for developing applications powered by language models.
- It's offered in Python and Javascript, providing an abstraction layer that integrates numerous libraries.
- It's also possible to deploy a built LangChain as an API.
**LlamaIndex** (MIT)
https://github.com/run-llama/llama_index
- It's a framework specialized in data for LLM applications.
- It provides data connectors for collecting existing data sources and formats (APIs, PDFs, documents, SQL, etc.).
- It offers ways to structure data (indexes, graphs) so that it can be easily used by LLMs.
**Haystack** (Apache-2.0)
https://github.com/deepset-ai/haystack
- It's an LLM framework for easy construction of Retrieval Augmented Generation (RAG), document search, question answering, and answer generation.
- It's built based on the pipeline concept.
**Flowise** (Apache-2.0)
https://github.com/FlowiseAI/Flowise
- It allows you to build customized LLM flows by dragging and dropping UI elements.
**LangFlow** (MIT)
https://github.com/logspace-ai/langflow
- It makes it easy to experiment with and prototype LangChain pipelines.
- It runs using the CLI and also supports deploying LangFlow on Google Cloud Platform (GCP).
**Spring AI** (Apache-2.0)
https://github.com/spring-projects/spring-ai
- An AI framework provided by Spring Framework (currently in snapshot status)
- It supports API integration based on OpenAI and MS Azure, and provides an abstraction layer.
- The goal is to implement AI features more easily and with greater scalability using AI Templates.
**Data Juicer** (Apache-2.0)
https://github.com/alibaba/data-juicer
- It's an open-source project released by Alibaba, serving as a one-stop data processing system for LLMs.
- It offers a systematic library composed of over 20 reusable configuration recipes, over 50 core OPs, and a feature-rich dedicated toolkit.
- Its automated report generation feature enables in-depth data analysis, leading to a deeper understanding of the datasets.
Comments0