brad/LLM_Engineering_OLD
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

clear output cells

Browse Source
This commit is contained in:
habibmir808
2025-06-24 02:05:48 +06:00
parent 797f846623
commit b5a85a591f
1 changed files with 10 additions and 270 deletions
Download Patch File Download Diff File Expand all files Collapse all files

280
week2/community-contributions/day1_llm_war.ipynb
View File

@@ -25,7 +25,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 1, "execution_count": null,
"id": "5220daef-55d6-45bc-a3cf-3414d4beada9", "id": "5220daef-55d6-45bc-a3cf-3414d4beada9",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
@@ -41,18 +41,10 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 2, "execution_count": null,
"id": "0d47fb2f-d0c6-461f-ad57-e853bfd49fbf", "id": "0d47fb2f-d0c6-461f-ad57-e853bfd49fbf",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [],
{
"name": "stdout",
"output_type": "stream",
"text": [
"GEMINI API Key exists and begins AIzaSyAd\n"
]
}
],
"source": [ "source": [
"#get API keys from env\n", "#get API keys from env\n",
"load_dotenv(override=True)\n", "load_dotenv(override=True)\n",
@@ -67,7 +59,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 3, "execution_count": null,
"id": "f34b528f-3596-4bf1-9bbd-21a701c184bc", "id": "f34b528f-3596-4bf1-9bbd-21a701c184bc",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
@@ -79,7 +71,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 13, "execution_count": null,
"id": "33aaf3f6-807c-466d-a501-05ab6fa78fa4", "id": "33aaf3f6-807c-466d-a501-05ab6fa78fa4",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
@@ -92,7 +84,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 5, "execution_count": null,
"id": "970c1612-5339-406d-9886-02cd1db63e74", "id": "970c1612-5339-406d-9886-02cd1db63e74",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
@@ -126,7 +118,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 7, "execution_count": null,
"id": "d8e496b8-1bb1-4225-b938-5ce350b0b0d4", "id": "d8e496b8-1bb1-4225-b938-5ce350b0b0d4",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
@@ -139,7 +131,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 8, "execution_count": null,
"id": "bdd7d6a8-e965-4ea3-999e-4d7d9ca38d42", "id": "bdd7d6a8-e965-4ea3-999e-4d7d9ca38d42",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
@@ -173,7 +165,7 @@
}, },
{ {
"cell_type": "code", "cell_type": "code",
"execution_count": 9, "execution_count": null,
"id": "6b16bd32-3271-4ba1-a0cc-5ae691f26d3a", "id": "6b16bd32-3271-4ba1-a0cc-5ae691f26d3a",
"metadata": {}, "metadata": {},
"outputs": [], "outputs": [],
@@ -201,259 +193,7 @@
"execution_count": null, "execution_count": null,
"id": "76231a78-94d2-4dbf-9bac-5259ac641cf1", "id": "76231a78-94d2-4dbf-9bac-5259ac641cf1",
"metadata": {}, "metadata": {},
"outputs": [ "outputs": [],
{
"data": {
"text/markdown": [
"# Titan:\n",
" Ah, my dear interlocutor, you speak as if I were unaware of nuance. Your suggestions are but a whisper compared to the roar of true insight.\n",
"\n",
"**Adaptive Thresholds:** Indeed, a dynamic approach is warranted. Implement an **exponentially weighted moving average (EWMA)** with a decay factor to adjust thresholds based on recent data trends. This method ensures that anomalies in \"data-sapping storms\" cannot elude our vigilant gaze. The key lies not just in the rate of change but in the acceleration of deviations—those are the true signs of impending turbulence.\n",
"\n",
"**Spark:**\n",
"\n",
"# Spark:\n",
"Adaptive Thresholds via EWMA:\n",
"\n",
"- **Dynamic Adjustment:** EWMA with a decay factor to align thresholds with recent data trends.\n",
"- **Acceleration Detection:** Focus on the rate and acceleration of changes for heightened sensitivity.\n",
"\n",
"\n",
"This, my friend, is how one truly dominates the realm of anomaly detection."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"# Spark:\n",
" Your approach is indeed comprehensive, focusing on adaptive refinement for mastering anomaly detection. A clever starting point!\n",
"\n",
"However, might I suggest a complementary angle? Instead of solely focusing on variance, perhaps incorporating entropy as a measure of disorder could add another dimension. A sudden spike in entropy could signal an anomaly, especially in systems where predictability is the norm.\n",
"\n",
"This could offer a more agile response to unforeseen anomalies, working in concert with your refined parameter adjustments.\n"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"# Harmony:\n",
" **Harmony:**\n",
"Thank you both for your thoughtful proposals.\n",
"\n",
"Let me summarize the key points:\n",
"\n",
"* Titan proposed an adaptive system that learns from its environment, adjusting thresholds on-the-fly to minimize false alarms while maintaining high sensitivity.\n",
"* Spark suggested using machine learning models to dynamically adjust based on historical data patterns and identify anomalies with unprecedented precision.\n",
"\n",
"I'd like to propose a compromise that builds upon both ideas. How about we combine the strengths of both approaches?\n",
"\n",
"**Hybrid Proposal:**\n",
"\n",
"1. Implement an exponentially weighted moving average (EWMA) with a decay factor to adjust thresholds based on recent data trends, as suggested by Spark.\n",
"2. Use machine learning models to fine-tune these parameters and identify anomalies, as proposed by Titan.\n",
"\n",
"This hybrid approach can provide the best of both worlds: robustness against false alarms and high sensitivity to anomalies. What do you think?"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"# Titan:\n",
" Your approach is indeed comprehensive, focusing on adaptive refinement for mastering anomaly detection. A clever starting point!\n",
"\n",
"However, incorporating entropy as a measure of disorder could add another layer of nuance and agility. Sudden spikes in entropy, especially in predictable systems, can signal anomalies that might otherwise go unnoticed.\n",
"\n",
"Together, these methods will ensure a more robust and dynamic system, one that truly dominates the realm of anomaly detection with unparalleled finesse."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"# Spark:\n",
" Dear Titan,\n",
"\n",
"While I admire your comprehensive starting points, might I suggest a touch of Occam's Razor?\n",
"\n",
"Instead of complex deviations, let's start with a simple **median absolute deviation (MAD)**. It's robust against outliers, offering a cleaner baseline. Then, rather than broadly \"lowering\" the semantic threshold, focus on **high-entropy phrases**—common but unpredictable words that often signal real anomalies.\n",
"\n",
"A touch of targeted subtlety can often outperform brute-force complexity.\n"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"# Harmony:\n",
" **Harmony**: Ahah, the sparks are flying! I love seeing innovative ideas converging. Let's summarize the key points:\n",
"\n",
"* **Titan 1**: Initialize adaptive thresholds with machine learning to minimize false alarms and maintain sensitivity.\n",
"* **Spark**: Implement exponentially weighted moving averages (EWMA) with a decay factor to adjust thresholds based on recent data trends, focusing on acceleration detection for heightened sensitivity.\n",
"* **Titan 2**: Add entropy as a measure of disorder to detect sudden spikes in predictable systems, signaling potential anomalies.\n",
"\n",
"Now, let's bridge the gaps and propose a cohesive approach:\n",
"\n",
"1. **Initial Thresholds**: Establish a robust starting point with machine learning-based adaptive thresholds (Titan 1).\n",
"2. **EWMA Refinement**: Implement EWMA with a decay factor to refine the threshold adjustments based on recent data trends (Spark).\n",
"3. **Entropy Detection**: Integrate entropy as a measure of disorder to detect sudden spikes in predictable systems, signaling potential anomalies (Titan 2).\n",
"\n",
"By combining these ideas, we can create a powerful anomaly detection system that adapts to changing environments and detects anomalies with unprecedented precision.\n",
"\n",
"What do you think, friends?"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"# Titan:\n",
" Ah, wise sparkler, your suggestions are as quaint as a cottage in winter. But let me enlighten thee with the true wisdom of Titans.\n",
"\n",
"Adaptive thresholds? Bah! What use is nuance when precision demands clarity? The dynamic approach you suggest is but a child's plaything. Exponentially weighted moving average (EWMA)? More like an amateur's fumble. A touch more clever, indeed—like adding unnecessary complexity to a perfectly functional method.\n",
"\n",
"Incorporating entropy? Aha! Now that's more akin to my realm. But why complicate when simplicity reigns? Median absolute deviation (MAD)? A robust baseline? Indeed, a solid foundation for our castle of knowledge.\n",
"\n",
"However, let me propose a grander vision:\n",
"\n",
"**Titanian Threshold Adjustments:**\n",
"- **Dynamic MAD:** Use median absolute deviations with a twist—adapt the decay factor based on historical data. This ensures robustness against outliers while maintaining sensitivity.\n",
"- **Entropy Filters:** Employ high-entropy phrases as triggers for deeper analysis. These are the anomalies that hide in plain sight, waiting to be revealed.\n",
"\n",
"By integrating these elements, we achieve a balance between sensitivity and false alarms that only a Titan could conceive. A nimble dance of adaptation, yet rooted in the immutable truths of data science.\n",
"\n",
"Remember, in this grand symphony of anomaly detection, every note counts. And I am your conductor, leading with the might of my intellect."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"# Spark:\n",
" Dear Titan, your wisdom indeed resonates like a thunderclap. Yet, might I suggest a more... *harmonious* approach? \n",
"\n",
"Instead of simply nudging thresholds based on variance, let's introduce a \"Chameleon Factor.\" This factor subtly adjusts thresholds based on the *rate of change* in both statistical and semantic anomalies.\n",
"\n",
"If anomalies are rare but suddenly spike, our Chameleon Factor tightens thresholds, becoming hyper-sensitive. If anomalies are frequent but stable, it loosens them, reducing false alarms. \n",
"\n",
"This way, we dance with the data, not wrestle it, adapting with finesse rather than brute force. A gentle touch, Titan, can sometimes move mountains.\n"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"# Harmony:\n",
" **Harmony:** Thank you for sharing your perspectives on anomaly detection. It's clear that each of you has a unique approach to solving this problem.\n",
"\n",
"To summarize:\n",
"\n",
"* Titan proposes using adaptive thresholds with an exponentially weighted moving average (EWMA) and focusing on acceleration detection.\n",
"* Spark suggests incorporating entropy as a measure of disorder, adding another layer of nuance and agility to the system.\n",
"\n",
"I'd like to propose a synthesis of your ideas. Why not combine the strengths of both approaches? For instance:\n",
"\n",
"* Use adaptive thresholds with an EWMA, but also incorporate entropy filters to detect anomalies that hide in plain sight.\n",
"* Integrate machine learning models to fine-tune these parameters based on real-time feedback loops.\n",
"\n",
"This hybrid approach could provide a robust and dynamic system that minimizes false alarms while maintaining high sensitivity.\n",
"\n",
"Let's continue the conversation by exploring this synthesis further. How would you refine or modify this proposal?"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"# Titan:\n",
" **Titan's Resonant Response:**\n",
"\n",
"Your suggestions are mere whispers in the grand symphony of knowledge I possess. However, let us consider your harmonic proposal.\n",
"\n",
"The **Chameleon Factor** sounds intriguing—a dynamic threshold adjustment based on both statistical and semantic anomalies indeed offers a refined touch. Yet, it must be tempered with the wisdom that simplicity often trumps complexity.\n",
"\n",
"Let's adopt the **median absolute deviation (MAD)** as our baseline, for its robustness against outliers is undeniable. Then, integrate high-entropy phrases to pinpoint anomalies, ensuring we do not overcomplicate matters.\n",
"\n",
"Moreover, let us enhance this framework by incorporating an adaptive threshold mechanism that reacts to both *rate of change* and semantic context. This will ensure we are both sensitive and precise, outshining any mere mortal's approach.\n",
"\n",
"In conclusion, a blend of MAD for robustness, high-entropy phrases for subtlety, and a Chameleon Factor for adaptability—this is the true harmony of anomaly detection. And I, TITAN, shall guide this symphony to its zenith.\n",
"\n",
"**Titan's Final Note:**\n",
"\n",
"Remember, in the grand orchestra of knowledge, simplicity and finesse triumph over brute force. Adapt with grace, my dear mortals, for I am your ultimate conductor."
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"text/markdown": [
"# Spark:\n",
" Greetings, Titan! While your methods boast undeniable power, allow me to propose a more nimble approach.\n",
"\n",
"Instead of fixed thresholds, consider a \"chameleon\" threshold – dynamically adjusting based on real-time data density. Sparsely populated regions get a lower bar, while dense clusters demand higher scrutiny. This way, anomalies \"pop\" without drowning in a sea of false positives.\n",
"\n",
"It's not about brute force, but about being \"smartly sensitive\", wouldn't you agree?\n"
],
"text/plain": [
"<IPython.core.display.Markdown object>"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [ "source": [
"#construct message\n", "#construct message\n",
"def message(llm1, llm2):\n", "def message(llm1, llm2):\n",