Datatype Limitation
Author Message
Tech80inc

 Posted: Mon Aug 08 12:45:35 CDT 2005 Top
 Visual Basic >> Datatype Limitation Hi, I used Double var to get the following exp claculated, 7064.93105099284 * (1 - 2 / 100) and I got the result as, 6923.63242997298. But When I used Calculator to do the evaluation of the same exp, I got 6923.6324299729832, where I got 2 more digits in the decimal part. No my question is whether this is a limitation of VB or is there a solution for the same. Jiju Joseph Visual Studio322

Rick

 Posted: Mon Aug 08 12:45:35 CDT 2005 Top
 Visual Basic >> Datatype Limitation > I used Double var to get the following exp claculated, > 7064.93105099284 * (1 - 2 / 100) and I got the result as, > 6923.63242997298. But When I used Calculator to do the evaluation of the same > exp, I got > 6923.6324299729832, where I got 2 more digits in the decimal part. > > No my question is whether this is a limitation of VB or is there a > solution for the same. It is not a limitation of VB; but, rather, a limitation of the data type you chose to use to hold your number. A Double can only hold (display actually), 15 significant digits, so that's what you got. IF you really need ALL the decimal digits for your program (hard to imagine any program where numbers in the 16th and 17th decimal position will actually matter significantly in your calculations), then you will have to use a Variant and convert at least one of your numbers to the Decimal data type. Debug.Print CDec(7064.93105099284) * (1 - 2 / 100) The Decimal data type can work up to 28 or 29 significant digits; however, there are limitation to this method if you plan on chaining the answer to other calculations. There is no overflow protection and built-in VB functions (such as Sin) will round your number down to the Double data type it is designed to work with. Personally, I'd just stick to the values the Double provides for you and leave it at that. Rick

Ralph

 Posted: Mon Aug 08 12:53:38 CDT 2005 Top
 Visual Basic >> Datatype Limitation > > I used Double var to get the following exp claculated, > > 7064.93105099284 * (1 - 2 / 100) and I got the result as, > > 6923.63242997298. But When I used Calculator to do the evaluation of > the same > > exp, I got > > 6923.6324299729832, where I got 2 more digits in the decimal part. > > > > No my question is whether this is a limitation of VB or is there a > > solution for the same. > > It is not a limitation of VB; but, rather, a limitation of the data type > you chose to use to hold your number. A Double can only hold (display > actually), 15 significant digits, so that's what you got. IF you really > need ALL the decimal digits for your program (hard to imagine any > program where numbers in the 16th and 17th decimal position will > actually matter significantly in your calculations), then you will have > to use a Variant and convert at least one of your numbers to the Decimal > data type. > > Debug.Print CDec(7064.93105099284) * (1 - 2 / 100) > > The Decimal data type can work up to 28 or 29 significant digits; > however, there are limitation to this method if you plan on chaining the > answer to other calculations. There is no overflow protection and > built-in VB functions (such as Sin) will round your number down to the > Double data type it is designed to work with. Personally, I'd just stick > to the values the Double provides for you and leave it at that. > > Rick > Rick, I wonder if it useful to occasionally remind programmers that NASA went to the moon and back with pi never figured beyond 3 places and no calculations over 4. -ralph