brad/LLM_Engineering_OLD
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Launching refreshed version of LLM Engineering weeks 1-4 - see README

Browse Source
This commit is contained in:
Edward Donner
2025-10-11 15:58:39 -04:00
parent 3286cfb395
commit c7257b9ae6
68 changed files with 16583 additions and 3756 deletions
Download Patch File Download Diff File Expand all files Collapse all files

466
guides/09_ai_apis_and_ollama.ipynb Normal file
View File

@@ -0,0 +1,466 @@
{
"cells": [
{
"cell_type": "markdown",
"metadata": {},
"source": [
"# LLM APIs and Ollama - beyond OpenAI\n",
"\n",
"_IMPORTANT: If you're not as familiar with APIs in general, and with Environment Variables on your PC or Mac, please review the APIs section in Guide 4 Technical Foundations before proceeding with this guide (topics 3 and 5 in Guide 4)._\n",
"\n",
"## Crucial context for using models other than OpenAI - please read this first!\n",
"\n",
"Throughout the course, we use APIs for connecting with the strongest LLMs on the planet.\n",
"\n",
"The companies behind these LLMs, such as OpenAI, Anthropic, Google and DeepSeek, have built web endpoints. You call their models by making an HTTP request to a Web Address and passing in all the information about your prompts.\n",
"\n",
"But it would be painful if we needed to build HTTP requests every time we wanted to call an API.\n",
"\n",
"To make this simple, the team at OpenAI wrote a python utility known as a \"Python Client Library\" which wraps the HTTP call. So you write python code and it calls the web.\n",
"\n",
"And THAT is what the library `openai` is.\n",
"\n",
"### What is the `openai` python client library\n",
"\n",
"It is:\n",
"- A lightweight python utility\n",
"- Turns your python requests into an HTTP call\n",
"- Converts the results coming back from the HTTP call into python objects\n",
"\n",
"### What it is NOT\n",
"\n",
"- It's not got any code to actually run a Large Language Model! No GPT code! It just makes a web request\n",
"- There's no scientific computing code, and nothing particularly specialized for OpenAI\n",
"\n",
"### How to use it:\n",
"\n",
"```python\n",
"# Create an OpenAI python client for making web calls to OpenAI\n",
"openai = OpenAI()\n",
"\n",
"# Make the call\n",
"response = openai.chat.completions.create(model=\"gpt-4.1-mini\", messages=[{\"role\":\"user\", \"content\": \"what is 2+2?\"}])\n",
"\n",
"# Print the result\n",
"print(response.choices[0].message.content)\n",
"```\n",
"\n",
"### What does this do\n",
"\n",
"When you make the python call: `openai.chat.completions.create()` \n",
"It simply makes a web request to this url: `https://api.openai.com/v1/chat/completions` \n",
"And it converts the response to python objects.\n",
"\n",
"That's it.\n",
"\n",
"Here's the API documentation if you make [direct web HTTP calls](https://platform.openai.com/docs/guides/text?api-mode=chat&lang=curl) \n",
"And here's the same API documentation if you use the [Python Client Library](https://platform.openai.com/docs/guides/text?api-mode=chat&lang=python)\n",
"\n",
"## With that context - how do I use other LLMs?\n",
"\n",
"It turns out - it's super easy!\n",
"\n",
"All the other major LLMs have API endpoints that are compatible with OpenAI.\n",
"\n",
"And so OpenAI did everyone a favor: they said, hey look - you can all use our utility for converting python to web requests. We'll allow you to change the utility from calling `https://api.openai/com/v1` to calling any web address that you specify.\n",
"\n",
"And so you can use the OpenAI utility even for calling models that are NOT OpenAI, like this:\n",
"\n",
"`not_actually_openai = OpenAI(base_url=\"https://somewhere.completely.different/\", api_key=\"another_providers_key\")`\n",
"\n",
"It's important to appreciate that this OpenAI code is just a utility for making HTTP calls to endpoints. So even though we're using code from the OpenAI team, we can use it to call models other than OpenAI.\n",
"\n",
"Here are all the OpenAI-compatible endpoints from the major providers. It even includes using Ollama, locally. Ollama provides an endpoint on your local machine, and they made it OpenAI compatible too - very convenient.\n",
"\n",
"```python\n",
"ANTHROPIC_BASE_URL = \"https://api.anthropic.com/v1/\"\n",
"DEEPSEEK_BASE_URL = \"https://api.deepseek.com/v1\"\n",
"GEMINI_BASE_URL = \"https://generativelanguage.googleapis.com/v1beta/openai/\"\n",
"GROK_BASE_URL = \"https://api.x.ai/v1\"\n",
"GROQ_BASE_URL = \"https://api.groq.com/openai/v1\"\n",
"OPENROUTER_BASE_URL = \"https://openrouter.ai/api/v1\"\n",
"OLLAMA_BASE_URL = \"http://localhost:11434/v1\"\n",
"```\n",
"\n",
"## Here are examples for Gemini, DeepSeek, Ollama and OpenRouter\n",
"\n",
"### Example 1: Using Gemini instead of OpenAI\n",
"\n",
"1. Visit Google Studio to set up an account: https://aistudio.google.com/ \n",
"2. Add your key as GOOGLE_API_KEY to your `.env` \n",
"3. Also add it a second time as GEMINI_API_KEY to your `.env` - this will be helpful later.\n",
"\n",
"Then:\n",
"\n",
"```python\n",
"import os\n",
"from openai import OpenAI\n",
"from dotenv import load_dotenv\n",
"load_dotenv(override=True)\n",
"\n",
"GEMINI_BASE_URL = \"https://generativelanguage.googleapis.com/v1beta/openai/\"\n",
"google_api_key = os.getenv(\"GOOGLE_API_KEY\")\n",
"gemini = OpenAI(base_url=GEMINI_BASE_URL, api_key=google_api_key)\n",
"response = gemini.chat.completions.create(model=\"gemini-2.5-flash-preview-05-20\", messages=[{\"role\":\"user\", \"content\": \"what is 2+2?\"}])\n",
"print(response.choices[0].message.content)\n",
"```\n",
"\n",
"### Example 2: Using DeepSeek API instead of OpenAI (cheap, and only $2 upfront)\n",
"\n",
"1. Visit DeepSeek API to set up an account: https://platform.deepseek.com/ \n",
"2. You will need to add an initial $2 minimum balance. \n",
"3. Add your key as DEEPSEEK_API_KEY to your `.env` \n",
"\n",
"Then:\n",
"\n",
"```python\n",
"import os\n",
"from openai import OpenAI\n",
"from dotenv import load_dotenv\n",
"load_dotenv(override=True)\n",
"\n",
"DEEPSEEK_BASE_URL = \"https://api.deepseek.com/v1\"\n",
"deepseek_api_key = os.getenv(\"DEEPSEEK_API_KEY\")\n",
"deepseek = OpenAI(base_url=DEEPSEEK_BASE_URL, api_key=deepseek_api_key)\n",
"response = deepseek.chat.completions.create(model=\"deepseek-chat\", messages=[{\"role\":\"user\", \"content\": \"what is 2+2?\"}])\n",
"print(response.choices[0].message.content)\n",
"```\n",
"\n",
"### Example 3: Using Ollama to be free and local instead of OpenAI\n",
"\n",
"Ollama allows you to run models locally; it provides an OpenAI compatible API on your machine. \n",
"There's no API key for Ollama; there's no third party with your credit card, so no need for any kind of key.\n",
"\n",
"1. If you're new to Ollama, install it by following the instructions here: https://ollama.com \n",
"2. Then in a Cursor Terminal, do `ollama run llama3.2` to chat with Llama 3.2 \n",
"BEWARE: do not use llama3.3 or llama4 - these are massive models not designed for home computing! They will fill up your disk. \n",
"\n",
"Then:\n",
"\n",
"```python\n",
"!ollama pull llama3.2\n",
"\n",
"from openai import OpenAI\n",
"\n",
"OLLAMA_BASE_URL = \"http://localhost:11434/v1\"\n",
"ollama = OpenAI(base_url=OLLAMA_BASE_URL, api_key=\"anything\")\n",
"response = ollama.chat.completions.create(model=\"llama3.2\", messages=[{\"role\":\"user\", \"content\": \"what is 2+2?\"}])\n",
"print(response.choices[0].message.content)\n",
"```\n",
"\n",
"### Example 4: Using the popular service [OpenRouter](https://openrouter.ai) which has an easier billing process instead of OpenAI\n",
"\n",
"OpenRouter is very convenient: it gives you free access to many models, and easy access with small upfront to paid models.\n",
"\n",
"1. Sign up at https://openrouter.ai\n",
"2. Add the minimum upfront balance as needed\n",
"3. Add your key as OPENROUTER_API_KEY to your `.env` file\n",
"\n",
"Then:\n",
"\n",
"```python\n",
"import os\n",
"from openai import OpenAI\n",
"from dotenv import load_dotenv\n",
"load_dotenv(override=True)\n",
"\n",
"OPENROUTER_BASE_URL = \"https://openrouter.ai/api/v1\"\n",
"openrouter_api_key = os.getenv(\"OPENROUTER_API_KEY\")\n",
"openrouter = OpenAI(base_url=OPENROUTER_BASE_URL, api_key=openrouter_api_key)\n",
"response = openrouter.chat.completions.create(model=\"openai/gpt-4.1-nano\", messages=[{\"role\":\"user\", \"content\": \"what is 2+2?\"}])\n",
"print(response.choices[0].message.content)\n",
"```\n",
"\n",
"\n",
"### Using different API providers with Agent Frameworks\n",
"\n",
"The Agent Frameworks make it easy to switch between these providers. You can switch LLMs and pick different ones at any point in the course. There are more notes below on each of them. For OpenAI Agents SDK, see a section later in this notebook. For CrewAI, we cover it on the course, but it's easy: just use the full path to the model that LiteLLM expects."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Costs of APIs\n",
"\n",
"The cost of each API call is very low indeed - most calls to models we use on this course are fractions of cents.\n",
"\n",
"But it's extremely important to note:\n",
"\n",
"1. A complex Agentic project could involve many LLM calls - perhaps 20-30 - and so it can add up. It's important to set limits and monitor usage.\n",
"\n",
"2. With Agentic AI, there is a risk of Agents getting into a loop or carrying out more processing than intended. You should monitor your API usage, and never put more budget than you are comfortable with. Some APIs have an \"auto-refill\" setting that can charge automatically to your card - I strongly recommend you keep this off.\n",
"\n",
"3. You should only spend what you are comfortable with. There is a free alternative in Ollama that you can use as a replacement if you wish. DeepSeek, Gemini 2.5 Flash and gpt-4.1-nano are significantly cheaper.\n",
"\n",
"Keep in mind that these LLM calls typically involve trillions of floating point calculations - someone has to pay the electricity bills!"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Ollama: Free alternative to Paid APIs (but please see Warning about llama version)\n",
"\n",
"Ollama is a product that runs locally on your machine. It can run open-source models, and it provides an API endpoint on your computer that is compatible with OpenAI.\n",
"\n",
"First, download Ollama by visiting:\n",
"https://ollama.com\n",
"\n",
"Then from your Terminal in Cursor (View menu >> Terminal), run this command to download a model:\n",
"\n",
"```shell\n",
"ollama pull llama3.2\n",
"```\n",
"\n",
"WARNING: Be careful not to use llama3.3 or llama4 - these are much larger models that are not suitable for home computers.\n",
"\n",
"And now, any time that we have code like: \n",
"`openai = OpenAI()` \n",
"You can use this as a direct replacement: \n",
"`openai = OpenAI(base_url='http://localhost:11434/v1', api_key='ollama')` \n",
"And also replace model names like **gpt-4o-mini** with **llama3.2**. \n",
"\n",
"You don't need to put anything in your .env file for this; with Ollama, everything is running on your computer. You're not calling out to a third party on the cloud, nobody has your credit card details, so there's no need for a secret key! The code `api_key='ollama'` above is only required because the OpenAI client library expects an api_key to be passed in, but the value is ignored by Ollama.\n",
"\n",
"Below is a full example:\n",
"\n",
"```python\n",
"# You need to do this one time on your computer\n",
"!ollama pull llama3.2\n",
"\n",
"from openai import OpenAI\n",
"MODEL = \"llama3.2\"\n",
"openai = OpenAI(base_url=\"http://localhost:11434/v1\", api_key=\"ollama\")\n",
"\n",
"response = openai.chat.completions.create(\n",
" model=MODEL,\n",
" messages=[{\"role\": \"user\", \"content\": \"What is 2 + 2?\"}]\n",
")\n",
"\n",
"print(response.choices[0].message.content)\n",
"```\n",
"\n",
"You will need to make similar changes to use Ollama within any of the Agent Frameworks - you should be able to google for an exact example, or ask me."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### OpenRouter: Convenient gateway platform for OpenAI and others\n",
"\n",
"OpenRouter is a third party service that allows you to connect to a wide range of LLMs, including OpenAI.\n",
"\n",
"It's known for having a simpler billing process that may be easier for some countries outside the US.\n",
"\n",
"First, check out their website: \n",
"https://openrouter.ai/\n",
"\n",
"Then, take a peak at their quickstart: \n",
"https://openrouter.ai/docs/quickstart\n",
"\n",
"And add your key to your .env file: \n",
"```shell\n",
"OPENROUTER_API_KEY=sk-or....\n",
"```\n",
"\n",
"And now, any time you have code like this: \n",
"```python\n",
"MODEL = \"gpt-4o-mini\"\n",
"openai = OpenAI()\n",
"```\n",
"\n",
"You can replace it with code like this:\n",
"\n",
"```python\n",
"MODEL = \"openai/gpt-4o-mini\"\n",
"openrouter_api_key = os.getenv(\"OPENROUTER_API_KEY\")\n",
"openai = OpenAI(base_url=\"https://openrouter.ai/api/v1\", api_key=openrouter_api_key)\n",
"\n",
"response = openai.chat.completions.create(\n",
" model=MODEL,\n",
" messages=[{\"role\": \"user\", \"content\": \"What is 2 + 2?\"}]\n",
")\n",
"\n",
"print(response.choices[0].message.content)\n",
"```\n",
"\n",
"You will need to make similar changes to use OpenRouter within any of the Agent Frameworks - you should be able to google for an exact example, or ask me."
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## OpenAI Agents SDK - specific instructions\n",
"\n",
"With OpenAI Agents SDK (weeks 2 and 6), it's particularly easy to use any model provided by OpenAI themselves. Simply pass in the model name:\n",
"\n",
"`agent = Agent(name=\"Jokester\", instructions=\"You are a joke teller\", model=\"gpt-4o-mini\")`\n",
"\n",
"You can also substitute in any other provider with an OpenAI compatible API. You do it in 3 steps like this:\n",
"\n",
"```python\n",
"DEEPSEEK_BASE_URL = \"https://api.deepseek.com/v1\"\n",
"deepseek_client = AsyncOpenAI(base_url=DEEPSEEK_BASE_URL, api_key=deepseek_api_key)\n",
"deepseek_model = OpenAIChatCompletionsModel(model=\"deepseek-chat\", openai_client=deepseek_client)\n",
"```\n",
"\n",
"And then you simply provide this model when you create an Agent.\n",
"\n",
"`agent = Agent(name=\"Jokester\", instructions=\"You are a joke teller\", model=deepseek_model)`\n",
"\n",
"And you can use a similar approach for any other OpenAI compatible API, with the same 3 steps:\n",
"\n",
"```python\n",
"# extra imports\n",
"from agents import OpenAIChatCompletionsModel\n",
"from openai import AsyncOpenAI\n",
"\n",
"# Step 1: specify the base URL endpoints where the provider offers an OpenAI compatible API\n",
"GEMINI_BASE_URL = \"https://generativelanguage.googleapis.com/v1beta/openai/\"\n",
"GROK_BASE_URL = \"https://api.x.ai/v1\"\n",
"GROQ_BASE_URL = \"https://api.groq.com/openai/v1\"\n",
"OPENROUTER_BASE_URL = \"https://openrouter.ai/api/v1\"\n",
"OLLAMA_BASE_URL = \"http://localhost:11434/v1\"\n",
"\n",
"# Step 2: Create an AsyncOpenAI object for that endpoint\n",
"gemini_client = AsyncOpenAI(base_url=GEMINI_BASE_URL, api_key=google_api_key)\n",
"grok_client = AsyncOpenAI(base_url=GROK_BASE_URL, api_key=grok_api_key)\n",
"groq_client = AsyncOpenAI(base_url=GROQ_BASE_URL, api_key=groq_api_key)\n",
"openrouter_client = AsyncOpenAI(base_url=OPENROUTER_BASE_URL, api_key=openrouter_api_key)\n",
"ollama_client = AsyncOpenAI(base_url=OLLAMA_BASE_URL, api_key=\"ollama\")\n",
"\n",
"# Step 3: Create a model object to provide when creating an Agent\n",
"gemini_model = OpenAIChatCompletionsModel(model=\"gemini-2.5-flash\", openai_client=gemini_client)\n",
"grok_3_model = OpenAIChatCompletionsModel(model=\"grok-3-mini-beta\", openai_client=openrouter_client)\n",
"llama3_3_model = OpenAIChatCompletionsModel(model=\"llama-3.3-70b-versatile\", openai_client=groq_client)\n",
"grok_3_via_openrouter_model = OpenAIChatCompletionsModel(model=\"x-ai/grok-3-mini-beta\", openai_client=openrouter_client)\n",
"llama_3_2_local_model = OpenAIChatCompletionsModel(model=\"llama3.2\", openai_client=ollama_client)\n",
"```\n",
"\n",
"### To use Azure with OpenAI Agents SDK\n",
"\n",
"See instructions here: \n",
"https://techcommunity.microsoft.com/blog/azure-ai-services-blog/use-azure-openai-and-apim-with-the-openai-agents-sdk/4392537\n",
"\n",
"Such as this:\n",
"```python\n",
"from openai import AsyncAzureOpenAI\n",
"from agents import set_default_openai_client\n",
"from dotenv import load_dotenv\n",
"import os\n",
" \n",
"# Load environment variables\n",
"load_dotenv()\n",
" \n",
"# Create OpenAI client using Azure OpenAI\n",
"openai_client = AsyncAzureOpenAI(\n",
" api_key=os.getenv(\"AZURE_OPENAI_API_KEY\"),\n",
" api_version=os.getenv(\"AZURE_OPENAI_API_VERSION\"),\n",
" azure_endpoint=os.getenv(\"AZURE_OPENAI_ENDPOINT\"),\n",
" azure_deployment=os.getenv(\"AZURE_OPENAI_DEPLOYMENT\")\n",
")\n",
" \n",
"# Set the default OpenAI client for the Agents SDK\n",
"set_default_openai_client(openai_client)\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## CrewAI setup\n",
"\n",
"Here's Crew's docs for LLM connections with the model names to use for all models. As student Sadan S. pointed out (thank you!), it's worth knowing that for Google you need to use the environment variable `GEMINI_API_KEY` instead of `GOOGLE_API_KEY`:\n",
"\n",
"https://docs.crewai.com/concepts/llms\n",
"\n",
"And here's their tutorial with some more info:\n",
"\n",
"https://docs.crewai.com/how-to/llm-connections"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## LangGraph setup\n",
"\n",
"To use LangGraph with Ollama (and follow similar for other models): \n",
"https://python.langchain.com/docs/integrations/chat/ollama/#installation\n",
"\n",
"First add the package: \n",
"`uv add langchain-ollama`\n",
"\n",
"Then in the lab, make this replacement: \n",
"```python\n",
"from langchain_ollama import ChatOllama\n",
"# llm = ChatOpenAI(model=\"gpt-4o-mini\")\n",
"llm = ChatOllama(model=\"gemma3:4b\")\n",
"```\n",
"\n",
"And obviously run `!ollama pull gemma3:4b` (or whichever model) beforehand.\n",
"\n",
"Many thanks to Miroslav P. for adding this, and to Arvin F. for the question!\n",
"\n",
"## LangGraph with other models\n",
"\n",
"Just follow the same recipe as above, but use any of the models from here: \n",
"https://python.langchain.com/docs/integrations/chat/\n",
"\n"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## AutoGen with other models\n",
"\n",
"Here's another contribution from Miroslav P. (thank you!) for using Ollama + local models with AutoGen, and Miroslav has a great example showing gemma3 performing well.\n",
"\n",
"```python\n",
"# model_client = OpenAIChatCompletionClient(model=\"gpt-4o-mini\")\n",
" \n",
"from autogen_ext.models.ollama import OllamaChatCompletionClient\n",
" \n",
"model_client = OllamaChatCompletionClient(\n",
" model=\"gemma3:4b\",\n",
" model_info={\n",
" \"vision\": True,\n",
" \"function_calling\": False,\n",
" \"json_output\": True,\n",
" \"family\": \"unknown\",\n",
" },\n",
")\n",
"```"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"## Worth keeping in mind\n",
"\n",
"1. If you wish to use Ollama to run models locally, you may find that smaller models struggle with the more advanced projects. You'll need to experiment with different model sizes and capabilities, and plenty of patience may be needed to find something that works well. I expect several of our projects are too challenging for llama3.2. As an alternative, consider the free models on openrouter.ai, or the very cheap models that are almost free - like DeepSeek.\n",
"\n",
"2. Chat models often do better than Reasoning models because Reasoning models can \"over-think\" some assignments. It's important to experiment. Bigger isn't always better...\n",
"\n",
"3. It's confusing, but there are 2 different providers that sound similar! \n",
"- Grok is the LLM from Elon Musk's X\n",
"- Groq is a platform for fast inference of open source models\n",
"\n",
"A student pointed out to me that \"Groq\" came first!\n"
]
}
],
"metadata": {
"language_info": {
"name": "python"
}
},
"nbformat": 4,
"nbformat_minor": 2
}