Jo Van Hoey

Beginning x64 Assembly Programming

From Novice to AVX Professional

Jo Van Hoey
Hamme, Belgium
ISBN 978-1-4842-5075-4e-ISBN 978-1-4842-5076-1
https://doi.org/10.1007/978-1-4842-5076-1
© Jo Van Hoey 2019
This work is subject to copyright. All rights are reserved by the Publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed.
Trademarked names, logos, and images may appear in this book. Rather than use a trademark symbol with every occurrence of a trademarked name, logo, or image we use the names, logos, and images only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The use in this publication of trade names, trademarks, service marks, and similar terms, even if they are not identified as such, is not to be taken as an expression of opinion as to whether or not they are subject to proprietary rights.
While the advice and information in this book are believed to be true and accurate at the date of publication, neither the authors nor the editors nor the publisher can accept any legal responsibility for any errors or omissions that may be made. The publisher makes no warranty, express or implied, with respect to the material contained herein.
Distributed to the book trade worldwide by Springer Science+Business Media New York, 233 Spring Street, 6th Floor, New York, NY 10013. Phone 1-800-SPRINGER, fax (201) 348-4505, e-mail [email protected], or visit www.springeronline.com. Apress Media, LLC is a California LLC and the sole member (owner) is Springer Science + Business Media Finance Inc (SSBM Finance Inc). SSBM Finance Inc is a Delaware corporation.
Introduction

Learning to program in assembly can be frustrating, not in the least because it is an unforgiving language; the computer will “yell” at you on every possible occasion. And if it doesn’t, you may just have unknowingly introduced a hidden bug that will bite you later in your program or at execution time. On top of that, the learning curve is steep, the language is cryptic, the official Intel documentation is overwhelming, and the available development tools have their own peculiarities.

In this book, you will learn to program in assembly starting with simple programs and moving all the way up to Advanced Vector Extensions (AVX) programming. By the end of this book, you will be able to write and read assembly code, mix assembly with higher-generation languages, understand what AVX is, and more. The purpose of this book is to show you how to use assembly language instructions. This book is not about programming style or code performance optimization. After you have acquired basic knowledge of assembly, you can continue learning how to optimize your code. This book should not be the first book you read on programming; if you have never programmed before, put this book aside for a while and learn some basics of programming with a higher-level language such as C.

All the code used in this book can be accessed via the Download Source Code link at www.apress.com/9781484250754 . The code used in this book is kept as simple as possible, which means no graphical user interfaces or bells and whistles or error checking. Adding all these nice features would deviate our attention from the purpose: to learn assembly language.

The theory is kept to a strict minimum: a little bit on binary numbers, a short presentation of logical operators, and some limited linear algebra. And we stay far away from doing floating-point conversions. If you need to convert binary or hexadecimal numbers, find a web site that does that for you. Don’t waste your time doing the calculations manually. Stick to the purpose: learning assembly.

The assembly code is presented in complete programs so that you can test them on your computer, play with them, change them, break them....

We will also show you what tools can be used, how to use them, and the potential problems in those tools. Having the right tools is essential to overcoming the steep learning curve. At times we will point you to books, white papers, and web sites that can be useful or that give more details.

It is not our intention to give you a comprehensive course on all of the assembly instructions. That is impossible in one book (look at the size of the Intel manuals!). We will give you a taste of the main items so that you will have an idea about what is going on. If you work through this book, you will acquire the knowledge to investigate certain domains in more detail on your own. When you have finished this book, you will be able to study the Intel manuals and (try to) make sense of their content.

The majority of the book is dedicated to assembly on Linux, because it is the easiest platform to learn assembly language. At the end, we provide a number of chapters to get you on your way with assembly on Windows. You will see that once you have Linux assembly under your belt, it is much easier to take on Windows assembly.

There are a number of assemblers available for use with Intel processors, such as FASM, MASM, GAS, NASM, and YASM to name a few. We will use NASM as in this book, because it is multiplatform; it is available on Linux, Windows, and macOS. Also, it has a relatively large user base. But don’t worry, once you know one assembler, it will be easy to adopt another assembly’s “dialect.”

We have carefully written and tested the code used in this book. However, if there are any typos in the text or bugs in the programs, we do not take any responsibility. We blame them on our two cats, who love to walk over our keyboard while we are typing.

The ideas and opinions we present in this book are our own and don’t necessarily represent IBM’s positions, strategies, or opinions.

Before You Start
You should know some basic things before you start reading this book.

  • You should know how to install and manage virtualization software (VMware, VirtualBox, or similar) . If you don’t have a clue what that means, download the free Oracle VirtualBox software ( https://www.virtualbox.org ), install it, and learn to use it by installing, for example, Ubuntu Desktop Linux as a guest operating system (OS). Virtualization software allows you to install different guest operating systems on your main machine, and if you mess things up in the guest system, you can delete that guest system and reinstall it. Or if you have taken a snapshot, you can return to a previous version of your guest installation. In other words, there’s no harm to your main (host) operating system when experimenting. There are plenty of resources on the Internet explaining VirtualBox and other virtualization software solutions.

  • You should have basic knowledge of the Linux command-line interface (CLI) . We will be using Ubuntu Desktop here, and we will use the CLI all the time, starting in Chapter 1 . You can use another Linux distribution if you want, but make sure you can install the tools used in the book (NASM, GCC , GDB , SASM, and so on). The following is the basic knowledge you need: how to install the OS, how to install additional software, how to start a terminal with a command prompt, and how to create, move, copy, and delete directories and files at the CLI. You also need to know how to use the tar utility, grep , find , ls , time , and so on. You need to know how to start and use a text editor. No advanced Linux knowledge is required; you need only a basic knowledge of these tasks to follow the explanations in this book. If you do not know Linux, play around with it to get used to it. There are plenty of good, short, beginning tutorials available on the Internet (e.g. https://www.guru99.com/unix-linux-tutorial.html ). You will see that after you learned assembly on a Linux machine, learning assembly on another OS is not that difficult.

  • You should have some basic knowledge of the C programming language . We will use a couple of C functions to simplify the example assembly code. Also, we will show how to interface with a higher-level language such as C. If you do not know C and want to fully enjoy this book, take a couple of free introductory C courses at, for example, tutorialspoint.com . There’s no need to do the whole course; just take a look at a few programs in the language. You can always return later to find out more details.

Why Learn Assembly?

Learning assembly has several benefits.

  • You’ll learn how a CPU and memory works.

  • You’ll learn how a computer and operating system work together.

  • You’ll learn how high-level language compilers generate machine language, and that knowledge can help you to write more efficient code.

  • You will be better equipped to analyze bugs in your programs.

  • It is a lot of fun when you eventually get your program working.

  • And the reason I wrote this book: if you want to investigate malware, you have only the machine code, not the source code. With an understanding of assembly language, you will be able to analyze malware and take necessary actions and precautions.

The Intel Manuals

The Intel manuals contain everything you ever wanted to know about programming Intel processors. However, the information is hard to swallow for a beginner. When you are progressing in this book, you will see that the explanations in these Intel manuals will make gradually more sense to you. We will refer often to these massive volumes of information.

You can find the Intel manuals here:

https://software.intel.com/en-us/articles/intel-sdm

Just don’t print them—think about all the trees you would be destroying! Take a short look at the manuals to see how comprehensive, detailed, and formal they are. Learning assembly from these manuals would be very daunting. Of special interest to us will be Volume 2, where you will find detailed explanations about the assembly programming instructions.

You will find a useful source here: https://www.felixcloutier.com/x86/index.html . This site provides a list of all the instructions with a summary of how to use them. If the information provided here is not sufficient, you can always go back to the Intel manuals or your friend Google.

Table of Contents

Chapter 1:​ Your First Program 1
Edit, Assemble, Link, and Run (or Debug) 2
Structure of an Assembly Program 7
section .​data 7
section .​bss 9
section .​txt 9
Summary 12
Chapter 2:​ Binary Numbers, Hexadecimal Numbers, and Registers 13
A Short Course on Binary Numbers 13
Integers 15
Floating-Point Numbers 16
A Short Course on Registers 16
General-Purpose Registers 16
Instruction Pointer Register (rip) 18
Flag Register 18
xmm and ymm Registers 19
Summary 19
Chapter 3:​ Program Analysis with a Debugger:​ GDB 21
Start Debugging 21
Step It Up! 27
Some Additional GDB Commands 30
A Slightly Improved Version of hello, world 31
Summary 33
Chapter 4:​ Your Next Program:​ Alive and Kicking! 35
Analysis of the Alive Program 37
Printing 40
Summary 45
Chapter 5:​ Assembly Is Based on Logic 47
NOT 47
OR 47
XOR 48
AND 49
Summary 50
Chapter 6:​ Data Display Debugger 51
Working with DDD 51
Summary 55
Chapter 7:​ Jumping and Looping 57
Installing SimpleASM 57
Using SASM 57
Summary 66
Chapter 8:​ Memory 67
Exploring Memory 67
Summary 75
Chapter 9:​ Integer Arithmetic 77
Starting with Integer Arithmetic 77
Examining Arithmetic Instructions 81
Summary 85
Chapter 10:​ The Stack 87
Understanding the Stack 87
Keeping Track of the Stack 91
Summary 93
Chapter 11:​ Floating-Point Arithmetic 95
Single vs.​ Double Precision 95
Coding with Floating-Point Numbers 97
Summary 100
Chapter 12:​ Functions 101
Writing a Simple Function 101
More Functions 103
Summary 106
Chapter 13:​ Stack Alignment and Stack Frame 107
Stack Alignment 107
More on Stack Frames 110
Summary 111
Chapter 14:​ External Functions 113
Building and Linking Functions 113
Expanding the makefile 117
Summary 119
Chapter 15:​ Calling Conventions 121
Function Arguments 122
Stack Layout 125
Preserving Registers 129
Summary 131
Chapter 16:​ Bit Operations 133
Basics 133
Arithmetic 140
Summary 145
Chapter 17:​ Bit Manipulations 147
Other Ways to Modify Bits 147
The bitflags Variable 150
Summary 151
Chapter 18:​ Macros 153
Writing Macros 154
Using objdump 156
Summary 157
Chapter 19:​ Console I/​O 159
Working with I/​O 159
Dealing with Overflows 162
Summary 166
Chapter 20:​ File I/​O 167
Using syscalls 167
File Handling 168
Conditional Assembly 178
The File-Handling Instructions 179
Summary 180
Chapter 21:​ Command Line 181
Accessing Command-Line Arguments 181
Debugging the Command Line 183
Summary 184
Chapter 22:​ From C to Assembler 185
Writing the C Source File 185
Writing the Assembler Code 187
Summary 194
Chapter 23:​ Inline Assembly 195
Basic Inline 195
Extended Inline 198
Summary 201
Chapter 24:​ Strings 203
Moving Strings 203
Comparing and Scanning Strings 209
Summary 214
Chapter 25:​ Got Some ID?​ 215
Using cpuid 215
Using the test Instruction 218
Summary 220
Chapter 26:​ SIMD 221
Scalar Data and Packed Data 221
Unaligned and Aligned Data 223
Summary 224
Chapter 27:​ Watch Your MXCSR 225
Manipulating the mxcsr Bits 226
Analyzing the Program 234
Summary 235
Chapter 28:​ SSE Alignment 237
Unaligned Example 237
Aligned Example 241
Summary 245
Chapter 29:​ SSE Packed Integers 247
SSE Instructions for Integers 247
Analyzing the Code 250
Summary 250
Chapter 30:​ SSE String Manipulation 251
The imm8 Control Byte 252
Using the imm8 Control Byte 253
Bits 0 and 1 253
Bits 2 and 3 254
Bits 4 and 5 255
Bit 6 255
Bit 7 Reserved 256
The Flags 256
Summary 257
Chapter 31:​ Search for a Character 259
Determining the Length of a String 259
Searching in Strings 262
Summary 266
Chapter 32:​ Compare Strings 267
Implicit Length 267
Explicit Length 270
Summary 275
Chapter 33:​ Do the Shuffle! 277
A First Look at Shuffling 277
Shuffle Broadcast 284
Shuffle Reverse 285
Shuffle Rotate 286
Shuffle Bytes 287
Summary 288
Chapter 34:​ SSE String Masks 289
Searching for Characters 289
Searching for a Range of Characters 296
Searching for a Substring 301
Summary 306
Chapter 35:​ AVX 307
Test for AVX Support 307
Example AVX Program 310
Summary 315
Chapter 36:​ AVX Matrix Operations 317
Example Matrix Code 317
Matrix Print:​ printm4x4 328
Matrix Multiplication:​ multi4x4 329
Matrix Inversion:​ Inverse4x4 332
Caley-Hamilton Theorem 332
Leverrier Algorithm 333
The Code 333
Summary 337
Chapter 37:​ Matrix Transpose 339
Example Transposing Code 339
The Unpack Version 344
The Shuffle Version 348
Summary 352
Chapter 38:​ Performance Optimization 353
Transpose Computation Performance 353
Trace Computation Performance 362
Summary 369
Chapter 39:​ Hello, Windows World 371
Getting Started 371
Writing Some Code 374
Debugging 376
Syscalls 376
Summary 376
Chapter 40:​ Using the Windows API 377
Console Output 377
Building Windows 380
Summary 382
Chapter 41:​ Functions in Windows 383
Using More Than Four Arguments 383
Working with Floating Points 390
Summary 392
Chapter 42:​ Variadic Functions 393
Variadic Functions in Windows 393
Mixing Values 395
Summary 397
Chapter 43:​ Windows Files 399
Summary 403
Afterword:​ Where to Go from Here?​ 405
Index 407

About the Author and About the Technical Reviewer

About the Author

Jo Van Hoey
../images/483996_1_En_BookFrontmatter_Figb_HTML.jpg

has 40 years of experience in IT, consisting of diverse functions, multiple IT companies, and various computing platforms. He recently retired from IBM, where he was a mainframe software account manager. He has always been interested in IT security, and knowledge of assembly language is an essential skill in defending IT infrastructure against attacks and malware.

 

About the Technical Reviewer

Paul Cohen
../images/483996_1_En_BookFrontmatter_Figc_HTML.jpg

joined Intel Corporation during the very early days of the x86 architecture, starting with the 8086, and retired from Intel after 26 years in sales/marketing/management. He is currently partnered with Douglas Technology Group, focusing on the creation of technology books on behalf of Intel and other corporations. Paul also teaches a class that transforms middle and high school students into real, confident entrepreneurs, in conjunction with the Young Entrepreneurs Academy (YEA), and is a traffic commissioner for the City of Beaverton, Oregon, and on the board of directors of multiple nonprofit organizations.

 
..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset
18.117.78.237