Alain AIROM’s Post

There are multiple considerations that go into #LLM agent implementation and prompt engineering. In our latest blog post, we discuss how to auto-evaluate LLM Agents and customize these agents to better work with LLMs that fit your specific use case. Read more --> https://scl.ai/llm-agent

We're excited to introduce eSentire Labs and our first open-source project, the eSentire LLM Gateway!    Today, we’re announcing our new innovation hub, eSentire Labs – a place where innovation occurs within eSentire with the sole objective of evaluating, validating, and prototyping new project ideas that originate from teams across our organization. The first project we will be releasing is the eSentire LLM Gateway, an open-source implementation framework to help organizations scale the use of Generative AI securely. “We are in the midst of a technology revolution where Generative AI is democratizing innovation across organizations of all sizes. But having the right controls is imperative to tap into its power, securely,” said Alexander Feick, VP of eSentire Labs. “We are helping security practitioners take the first step to gaining visibility and control over this exciting innovation with the eSentire LLM Gateway.” Benefits of the eSentire LLM Gateway include: - Creates a protective layer between corporate data and open AI applications including ChatGPT - All LLM interactions are logged, enhancing monitoring & security control, while also surfacing core use cases from functional and power users - As an open-source framework, security practitioners can integrate and apply their own security controls including corporate policies, usage rules and prompts, and contribute suggestions to further collaboration with the broader cybersecurity community - Basic recommendations on how to visualize and track LLM usage are included within the eSentire LLM Gateway’s initial plug-ins Access our open-source LLM Gateway on GitHub:

Prompt engineering is a new space :) With the advent of LLMs, all I see is people talking about how to prompt in order to get the best results. Who knew that - there will be a time we will learn how to talk to Chat Bots :) I wish there is a prompt engineering for humans as well. Currently, when we ask questions and others don't answer properly, we blame the human answering (pretty much all the time). The art of how to formulate a question is also important, and a skill everyone needs to learn.

Some raw thoughts / learnings on LLM optimization after spending 2 months working with LLM teams on optimizing their infra stack. We're now working with 10-15 teams on optimizing their infra, and these are my key learnings: 1. Good model selection > fine tuning, 95% of the time The cases where fine tuning works really well are for things like text classification (for simple things) using 7b models. Fine tuning as an optimization is actually an edge case, not the main optimization. For *most* if not *all* use cases, the best optimization comes down to using a sonnet, mistral-medium, mistral-small or haiku foundation model that performs at GPT-4 level for a given task but is much faster, without a fine tune, and often beats a fine tune on cost and latency. This is enabled by good evals, not fine tuning infrastructure. 2. 50-60% of the teams using flyflow are building real-time voice apps Weird niche that we fell into, but very fun to be working with. A bunch of our early customers need low latency to build real time AI voice <> human applications. Over the weekend I'm working on doing a super good fine tune instruct model for these applications which we'll host at low latency. I think there's room in the market for a foundation model that handles human to human interactions via voice at super low latency, and this is an area we'll be exploring deeply. As always, if you're interested in what we're building and want to optimize your LLM infra stack feel free to reach out. You can either DM me or sign up for a demo at https://flyflow.dev

I thought this was a very intuitive guide to LLM technology https://lnkd.in/gUx4WpdR

Navigating the Shift to GPT-4: A Cautionary Tale and Key Learnings Some caution for people considering switching to the latest GPT-4 model. Considering that the API for the gpt-4-1106-preview, which is the GPT-4 Turbo model, is already out, I decided to give it a try to see if it could perform the task that the previous GPT-4 model did in my project. After replacing it, I found that it failed almost instantly. The output quality could have been higher, and it could not format the output text according to my specifications. I can still use it by adjusting the input, but it will take some time for me to format my prompt to match the quality of the previous model. Key Takeaways: Model Upgrade Complexity: Contrary to expectations set by OpenAI, upgrading to the latest GPT-4 model is complicated. The shift to the n ew model reduced performance and required extensive prompt tuning to restore functionality to previous levels. Read more about our experience here: https://lnkd.in/g8ffRKTV API Update Implications: The top-level API (like GPT-4 and GPT-3.5-turbo) defaults to the newest model. This can lead to performance dips when the underlying model is updated. Always specify your GPT version (e.g., gpt-4-0314) in your projects to avoid unexpected changes. Turbo Version Trade-offs: The Turbo versions, including GPT-4 Turbo, are compressed or quantized models, optimized for speed and cost-efficiency, at the expense of output quality.  When comparing prices, the input tokens for the new GPT4-Turbo API are 3x cheaper than those for the GPT4 model. Reducing costs is a welcome change for someone who builds products using GPT. However, it also means that the output quality will be significantly worse. ChatGPT 3.5-Turbo is a prime example, inferior to GPT3-based Code-Davinci-002. If you’re considering an upgrade to GPT-4 or its Turbo variant, these insights might save you time and effort. #openai #llm #largelanguagemodels #openaidevday #genai

Introducing...eSentire Labs and our first open-source project, the eSentire LLM Gateway! 🔬 👨🔬   Today, we’re announcing our new innovation hub, eSentire Labs – a place where innovation occurs within eSentire with the sole objective of evaluating, validating, and prototyping new project ideas that originate from teams across our organization. The first project we’re releasing? The eSentire LLM Gateway, an open-source implementation framework to help organizations scale the use of Generative AI securely. “We are in the midst of a technology revolution where Generative AI is democratizing innovation across organizations of all sizes. But having the right controls is imperative to tap into its power, securely,” said Alexander Feick, VP of eSentire Labs. “We are helping security practitioners take the first step to gaining visibility and control over this exciting innovation with the eSentire LLM Gateway.” Benefits of the eSentire LLM Gateway include: ✔ Creates a protective layer between corporate data and open AI applications including ChatGPT  ✔ All LLM interactions are logged, enhancing monitoring & security control, while also surfacing core use cases from functional and power users ✔ As an open-source framework, security practitioners can integrate and apply their own security controls including corporate policies, usage rules and prompts, and contribute suggestions to further collaboration with the broader cybersecurity community  ✔ Basic recommendations on how to visualize and track LLM usage are included within the eSentire LLM Gateway’s initial plug-ins Access our open-source LLM Gateway on GitHub: https://bit.ly/3QOlhxx